DOCUMENT RESUHE 



ED 306 803 



HE 022 464 



AUTHOR 
TITLE 

INSTITUTION 
PUB DATE 
NOTE 



PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



ABSTRACT 



Barak, Robert; And Others 

An Evaluation of the Decennial Review Process. 
Arizona Board of Regents, Phoenix. 
Nov 88 

7p.; In "The Arizona Board of Regents' Task Force on 
Excellence, Efficiency, and Competitiveness. Final 
Report and Working Papers." Volame One. For Volumes 
One and Two, see HE 022 446-447. For individual 
working papers, see HE 022 448-480 and HE 022 
482-501. 

Reports - Evaluative/Feasibility (142) 
MF01/PC01 Plus Postage. 

"College Programs; Competition; "Educational Quality; 
Efficiency; Excellence in Education; Higher 
Education; "Policy Formation; "Program Evaluation; 
Public Colleges; State Universities 
"Arizona; Arizona State University; Arizona Task 
Force on Excellence Efficency Compet; Northern 
Arizona University; University of Arizona 



Information on a review of the decennial review 
process required by the Arizona Board of Regents (ABOR) for every 
academic department in each Arizona university is presented as part 
of the final report by ABOR's Task Force on Excellence, Efficiency 
and Competitiveness. Revision in the review process and eight policy 
statements are discussed, and three characteristics of a robust 
program review are noted: they should occur on a regular schedule; 
they should be conducted in a systematic fashion; and they should be 
comprehensive. Eleven recommendations are made by the Task Force 
staff, including: (l) intervals between reviews should be reduced 
from 10 years to a period of from 5 to 7 years; (2) ABOR should more 
closely monitor reviews to ensure they are responsive to the stated 
requirements of their review policy; (3) each university should 
develop procedures and guidelines for meaningful program review; (4) 
procedures for the use of consultants should include greater 
objectivity in their selection; (5) procedures for the reviews should 
encourage active participation of department faculty; and (6) the 
reviews should be used by department chairs, deans, and vice 
presidents for planning and resource allocation. The rationale behind 
and responsibility for each recommendation are discussed. (SM) 
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"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



AZ Board of Regents 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) " 

INTRODUCTION 

"A program review is a formal and 
systematic process that examines 
programs periodically over time, checks 
for changes, and matches program 
characteristics with ongoing program 
needs" (Ewell, 1983). During the last 
decade, program review has become an 
important part of the planning process in 
higher education. About one-third of all 
institutions of higher education now have 
formal program review. More than half 
of these have been developed since 
1977 (Ewell, 1983). Given these facts, it 
appears that the Arizona Board of 
Regents (ABOR) was a pioneer in this 
effort. Since 1975, ABOR has had a 
policy requiring all academic 
departments located in Arizona 
universities to undergo a substantial 
review process every 10 years. The 
policy has undergone periodic revision 
and is outlined in a series of policy 
statements: 

1. The process of acadenic program 
review should be primarily internal to 
the university, should produce 
judgments regarding disciplines by 
colleagues, and should include an 
assessment of the performance and 



Robart Bank It Deputy Emcullvt Sacrataiy and Director of Acadtmle AfraJro 
and S m tfch, Iowa State Board of Raganta. 



REVIEW PROCESS 



ROBERT BARAK 
and 

THE TASK FORCE STAFF 



U3 DEPARTMENT OF EDUCATION 

Office of Educational Reaearch and improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

•GJtfis' document has been reproduced as 
received from the person or organization 
originating iL 

r Minor changes have been made to improve 
reproduction quality 

e Points of view or opinions stated in this docu- 
ment do not necessarily represent official 
OERI position or policy 



achievements of students and of the 
contributions of the discipline to other 
campus units. 

2. Four to eight academic departments 
should be reviewed each year, with 
the objective of a complete review of 
all departments over a ten-year 
period. 

3. An initial review (self-study) will be 
undertaken by faculty and 
administrators within each discipline. 
Undergraduate and graduate 
disciplines will also be reviewed by 
colleagues or administrators from 
outside the discipline, but within the 
university. Undergraduate disciplines 
may also be reviewed by external 
consultants. Graduate disciplines will 
be reviewed by external consultants. 

4. Although departments are the basic 
unit for review, it is expected than 
some programs and curricula will be 
reviewed at the college level. 

5. Budgetary needs for academic 
program reviews will be met by 
existing budgets of the universities. 

6. Reviews will cover the areas of 
academic degree programs, faculty, 
students, physical facilities, costs, 
other resources and production. 
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7. Summary reports will be submitted to 
the chair of the Academic Affairs 
Committee by the involved university 
administrator and the Regent's staff. 

8. The Academic Affairs Council may 
make recommendations to the 
university presenting the academic 
program review. All proposals for 
academic program or curriculum 
changes resulting from the reviews 
will be subject to the same approval 
procedures required for other 
proposals for changes. 

(Chapter II, Sec. 2-208, Board of 
Regents Policy Manual) 

Most would agree that the three 
characteristics of a robust program 
review are 1) they should occur on a 
regular schedule 2) they should be 
conducted in a systematic fashion and 
3) they should be comprehensive. 
Where the policies outlined by the Board 
are followed, robust reviews result. 
However, if all program reviews are to 
become serious ingredients in the 
planning process, the review process 
must avoid common pitfalls. Those 
pitfalls might include 1) problems with 
use of data, 2) problems in the conduct 
of the reviews, and 3) problems that 
have to do with whether or not the 
review results are used and how they 
are used. 

The Task Force staff undertook a review 
of the process for periodic program 
review with the aid of a consultant, Dr. 
Robert Barak, Associate Director, Iowa 
Board of Regents, a specialist in 
academic program reviews. Dr. Barak 
spent a total of four days on site visits to 
the ABOR office and to the institutions. 
The recommendations to be presented 
represent the combined efforts of Dr. 
Barak and the Task Force staff. They 



were formulated following an 
examination of written policy statements, 
guidelines, self studies, consultant 
reports and summary reports to the 
Board of Regents, in addition to 
discussions with ABOR staff, 
representatives of academic vice 
presidents, deans, departmental chairs 
and faculty at the three institutions. 

The current review process has many 
strengths including an excellent potential 
for strengthening academic programs, 
removing marginal or nonessential 
programs and increasing accountability 
to higher education constituencies. The 
review process is currently undergoing 
major revisions on all campuses with 
these goals in mind. However, if the 
goals are to be realized, there are certain 
shortcomings that need to be 
addressed. It is recognized that some 
of the recommendations which follow 
have already been implemented by 
some units. Nevertheless, the 
effectiveness of the process is uneven 
throughout the state. 

RECOMMENDATIONS 

o A statement of the goals and 
objectives of the academic program 
review effort should be incorporated 
into Board of Regents policy so that 
the Board's expectations for this 
effort are clear. The statement should 
emphasize the value of program 
review to the institution and to the 
department conducting the review. 

Rationale: 

The recommendation addresses the 
need for immediate, recognizable 
benefits at the institutional level and for 
greater incentives for full, conscientious 
involvement in the reviews. It should be 
emphasized that the reviews are done to 
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promote program improvement, assure 
maximum use of current resources and 
provide a basis for university planning. 
The purpose of the reviews appears to 
be unknown to many faculty, 
departmental level administrators, and 
deans. While some units and deans have 
used the information provided by the 
review process very effectively, the 
reviews are often viewed as an 
unnecessary burden placed on the 
institutions by the Board of Regents 
rather than as a means for program 
improvement. 

Responsibility: Board of Regents 

o Intervals between reviews should be 
reduced from ten years to a shorter 
period of from five to seven years. 

Rationale: 

Ten years is a very long time between 
reviews of program quality. Good 
programs have been known to 
deteriorate in just a few years, while 
others may emerge and not be fully 
recognized. The national average for 
periodic program reviews is five to six 
years. 

Responsibility: Board of Regents 

o Various internal and external review 
efforts should be integrated in order 
to reduce the overall burden on 
faculty and departmental 
administrators and to improve the 
chances of periodic program reviews 
being utilized. Accreditation reports 
normally should not be used as a 
substitute for periodic program 
reviews unless they meet all criteria 
for the review. 



Rationale: 

In some cases, institutions have been 
substituting accreditation reviews for the 
decennial review efforts. In other cases 
(for example, some programs at the 
medical school) units have escaped 
review altogether. There are some 
substantive differences between program 
reviews conducted by institutions and 
accreditation reviews conducted by 
outside special interest organizations. 
Most accreditation groups rarely address 
the issue of whether a program is 
needed in the first place, whereas an 
institutional review ought to consider this 
issue for each program area. In 
addition, some accreditation 
organizations do an excellent job of 
preparing site-visits, developing quality 
indicators and conducting the total 
accreditation review, while others are 
frequently found wanting. Because of 
quality differences in external 
accreditation processes, efforts should 
be made to ensure that external 
accreditation reviews meet all guidelines 
of internal program review as well as 
those required for the accreditation 
process if the review is to be substituted 
for the internal periodic program review. 

Responsibility: Board of Regents and 
universities 

o The Board of Regents should more 
closely monitor reviews to ensure 
that they are responsive to the stated 
requirements of their review policy. 

Rationale: 

The summary reports now submitted to 
the Board of Regents do not reflect the 
actual review process and contain little 
or no useful information for decision 
making. They should be discontinued 
in favor of an assessment of each review 
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by the Board of Regents staff. Staff 
reports to the Academic Affairs Council 
should accurately reflect the content of 
the reviews and the follow-up efforts 
required for recommendations and 
concerns. 

The Board of Regents and the 
universities should dearly define data to 
be included in the review process and 
develop standardized reporting 
methods. For example, current Regents' 
policy states that the reviews "should 
include an assessment of the 
performance and achievement of 
students." Assessment is uneven and, 
under most circumstances, self serving 
to the program under review. 
Consideration should be given to the 
development of standardized survey 
instruments with surveys being 
conducted by experienced individuals 
outside of the program under e/aluation. 

Responsibility: Board of Regents and 
universities 

o Each university should develop 
procedures and guidelines for 
meaningful program review. Board 
staff should periodically monitor the 
institutions to ensure that procedures 
and guidelines are in place and 
being used. 

Rationale: 

There appears to be a lack of familiarity 
with institutional guidelines b'/ some 
deans, departmental chairs and' faculty. 
Because program review processes are 
currently undergoing revision, the 
guidelines are frequently "drafts" with 
insufficient specificity to lead to 
consistent, comprehensive review of 
programs. As a result, considerable time 
is spent collecting information that is of 
little use in determining program quality. 



Responsibility: Universities 

o Policies and procedures for the 
reviews should emphasize a primary 
focus on program improvement and , 
a secondary focus on resource 
requests. The recommendations 
contained in the reviews should be 
used as a starting point for the next 
periodic review. 

Rationale: 

Content of the reviews are overly 
focused on resource acquisition at the 
expense of an analysis of program 
improvement. Almost all note the need 
for more faculty, higher salaries, 
more/better equipment and facilities. 
Reviews seem to be viewed only as 
opportunities to seek incremental 
resource increases. While this can and 
should be a part of the self-study where 
appropriate, resource acquisition should 
not be the overriding goal. 

In some instances, problems identified 
in the reviews, such as the lack of 
resources, are not addressed creatively 
because of inflexibility created by the 
perception (correct or incorrect) that 
more students mean more dollars. For 
example, some units will not propose to 
cap enrollments in order to reduce 
faculty workloads because it is perceived 
that support dollars are tied to the 
number of students enrolled. The 
guidelines should ask departments to 
review the program and suggest 
improvements given existing resources 
and faculty lines to achieve optimum 
quality. 

Responsibility: Universities 

o Procedures for the use of consultants 
should include greater objectivity in 
their selection. On-site visits and exit 
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interviews with department chairs and 
deans should be mandatory. 

Rationale: 

Soma aspects of existing guidelines 
impair review objectivity. For example, 
outside consultants are often picked 
exclusively from lists submitted by the 
department under review. In some 
instances, the outside consultants never 
visit the campus but merely respond to 
a written self-study prepared by the 
department. This results in an 
under-utilization of the consultants, a 
reduction in the objectivity of the 
reviews, and the loss of an opportunity 
to receive observations for program 
improvement which are often provided 
during a frank, confidential exit interview 
with key administrators. 

The possible inclusion of a "generalist" 
consultant^ non-peer to the department 
but knowledgeable about administration 
of reviews should be considered. 
Consideration should also be given to 
providing consultants with information on 
where a department under review fits 
into the overall mission and goals of the 
university and the college. This may 
help overcome the problem of 
consultants developing unrealistic 
recommendations for a given unit. 

Responsibility: Universities and 
departments 

o Procedures for the reviews should 
encourage active participation of 
department faculty. Each member 
should have the opportunity to review 
the completed self-study, consultant 
reports, and institutional oversight 
committee reports. 



Rationale: 

In some instances, faculty in a 
department being reviewed do not 
actively participate in the reviews and do 
not have an opportunity to see, much 
less discuss the review results. 

Responsibility: Universities and 
departments 

o Undergraduate and graduate 
programs should be given equal 
emphasis in the periodic program 
review. Restructuring of university 
oversight committees and a change in 
ABOR policy statement number 3 may 
be required to ensure than this 
occurs. 

Rationale: 

Undergraduate programs are sometimes 
ignored and are often superficially 
covered. In some cases this is true of 
both the self-studies and the consultant 
reports.This may be a result of the 
graduate level emphasis of the reviews, 
for in some instances the reviews are 
conducted under the general supervision 
of the graduate dean. ABOR policy 
statement number 3 can also be 
interpreted as emphasizing graduate 
rather than undergraduate programs. 

Responsibility: Universities and 
departments 

o The reviews should be used by 
department chairs, deans and vice 
presidents for planning and resource 
allocation. 

Rationale: 

While some periodic reviews represent 
some of the most extensive and best 
compilation of information on programs, 
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they appear to be used infrequently 
outside the department for planning and 
budgeting purposes. Follow-up on 
recommendations by the dean and vice 
presidential levels appears to be rare. 
Schedules of reviews need to mesh 
effectively with other elements of 
university strategic decision-making 
cycles for planning and budgeting. 

Responsibility: Universities 

o Financial, data gathering and 
analysis, and clerical assistance 
should be provided to departments 
undergoing reviews. 

Rationale: 

A comprehensive, effective, program 
review is a strain on a departmental 
operating budget. Financial incentives 
must be created to encourage 
departments to undertake certain 
activities to improve the quality of the 
review. For example, meaningful 
surveys are not inexpensive. For some 
units, the expense of such surveys 
would decimate the operations budget. 
Often departmental expertise is lacking 
for conducting surveys and interpreting 
the data. 

Responsibility: Board or Regents and 
universities 
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